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Single-particle dynamics in the 120° ordered antiferromagnetic state of the triangular-lattice Hub- 
bard model is studied using a physically transparent fluctuation approach in terms of multiple 
magnon emission and absorption processes within the noncrossing approximation. Hole and elec- 
tron spectral features are evaluated at intermediate U, and analyzed in terms of a competition be- 
tween the frustration- induced direct hopping and the virtual hopping. Finite U -induced competing 
interactions and frustration effects contributing through the magnon dispersion are also discussed. 
Finite contribution to self-energy correction from long-wavelength (Goldstone) modes, together with 
the high density of electron scattering states in the narrow, sharp peak in the upper band, result 
in strong fermion-magnon scattering leading to pronounced incoherent behaviour in the electron 
dynamics. The fluctuation-induced first-order metal-insulator transition due to vanishing band gap 
is also discussed. 
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I. INTRODUCTION 

There has been renewed interest in correlated elec- 
tron systems on triangular lattices, as evidenced by re- 
cent studies of antiferromagnetism, superconductivity 
and metal- insulator transition in the organic systems k — 
(BEDT - TTF) 2 X^ the discovery of superconductivity 
in Na x Co02.yH20, 3 the observation of low-temperature 
insulating phases in some \/3-adlayer structures such as 
K on Si[lll]^ and quasi two-dimensional 120° spin or- 
dering and spin- wave excitations in RbFe(Mo04)2 (Refs. 
5,6) and the multiferroic material HoMnOai^ 

The recent finding of finite {/-induced competing in- 
teractions and frustration in the 120° ordered antifer- 
romagnetic (AF) state of the Hubbard model on a tri- 
angular lattice^ resulting in vanishing spin stiffness at 
U i=s 6 and a magnetic instability towards a F-AF state 
at U ~ 7, both in the insulating state, adds a new 
dimension to the intrinsic geometric frustration of the 
triangular-lattice antiferromagnet. Indeed, strongly en- 
hanced quantum spin fluctuations associated with the 
magnetic instability may account for the absence of long- 
range magnetic order in the nearly isotropic organic an- 
tiferromagnet k- (BEDT-TTF) 2 Cu 2 (CN) 3 , as inferred 
from recent 1 H NMR and static susceptibility measure- 
ments down to 32 mK, well below the estimated ex- 
change constant J ~ 250 K, suggesting the realization 
of a quantum spin-liquid stated The realization of a 
non-magnetic insulator at intermediate U is also inter- 
esting as it allows, with decreasing U, for a Mott-type 
metal-insulator transition not accompanied by any mag- 
netic symmetry breaking, as seen in the layered system 
k - (BEDT - TTF) 2 Cu[N(CN) 2 ]Cl,±i and currently of 
much theoretical interest 

Recently, self-energy corrections due to multiple 
magnon processes in the AF state of the frustrated 
square-lattice t — i'-Hubbard model were evaluated us- 
ing a fluctuation approach. 14 Quasiparticle dispersion 



obtained with the same set of Hubbard model cuprate 
parameters as obtained from a recent magnon spectrum 
fit 15 was found to yield excellent agreement with ARPES 
data for Sr 2 Cu0 2 Cl 2 j 1 * thus providing a unified descrip- 
tion of magnetic and electronic excitations in cuprates. 

It is therefore interesting to examine and contrast self- 
energy corrections and quasiparticle behaviour in the 
120° ordered AF state of the triangular-lattice Hubbard 
model which involves non-collinear ordering, intrinsic ge- 
ometric frustration, and also finite (7-induced competing 
interactions and frustration. Indeed, we find that long- 
wavelength magnon modes yield finite contribution to the 
fermion-magnon scattering process, unlike the square- 
lattice case where this contribution was negligible. 14 

Frustration and spin fluctuations are involved in an 
interesting interplay with respect to quasiparticle be- 
haviour. As neighbouring spins are not antiparallel in 
triangular-lattice and frustrated square-lattice AF states, 
frustration results in an 0(t) or O(i') direct hopping 
in addition to the O(J) virtual hopping. Competi- 
tion between the two dispersions results in band broad- 
ening/narrowing, which has a dramatic effect on self- 
energy corrections due to significantly different density 
of fermion scattering states. Competition also results in 
a reduced band gap, thus bringing the system closer to 
a metal-insulator (MI) transition. The t — J model cal- 
culations do not involve this competition as the virtual 
hopping term is absent. 

In this paper, we examine the hole and electron spec- 
tral functions in terms of self-energy corrections in the 
120° ordered AF state of the Hubbard model on a tri- 
angular lattice. Hole and electron dynamics in an an- 
tiferromagnetic background is associated with multiple 
magnon emission and absorption processes correspond- 
ing to broken AF bonds, string states, and scrambling of 
the AF spin ordering. The fluctuation approach adopted 
in this paper in terms of a diagrammatical expansion pro- 
vides a physically transparent picture in which the hole 
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motion is renormalized due to the fluctuating transverse 
field associated with spin fluctuations. This fluctuation 
perspective is physically relevant in view of the substan- 
tial ordered-moment reduction, which approaches 59% in 
the strong-coupling (Heisenberg) limits 

Single-particle dynamics in the AF state of the 
triangular-lattice t — J model has been studied in the 
self-consistent Born approximation (SCBA)jiLi2*i2i exact 
diagonalizationjii and projection techniques ,22*21 One- 
electron density of states has been examined using the 
quantum Monte Carlo method^ showing a pseudogap 
development for intermediate U. 

We consider the Hubbard model 



(1) 



with nearest-neighbour (NN) hopping on a triangular lat- 
tice. The model has particle-hole symmetry under the 
transformation t — > —t. In this paper we consider the 
case of positive t and hole (electron) doping in the lower 
(upper) band; same result holds for negative t and elec- 
tron (hole) doping in the upper (lower) band. In the 
following we set the hopping energy t = 1. 

The organization of this paper is as follows. The three- 
sublattice basis is briefly reviewed in section II to intro- 
duce the notation and key features of the classical-level 
fcrmion dispersion. Transverse spin fluctuations are in- 
troduced in section III in terms of magnon amplitudes 
and energies. Intraband self-energy corrections due to 
multi-magnon processes are studied in section IV in the 
non-crossing approximation. Hole and electron spectral 
functions for the lower and upper bands are discussed in 
section V and conclusions are presented in section VI. 



II. THREE-SUBLATTICE REPRESENTATION 

While the spiral-state description applies only to Bra- 
vais lattices, the sublattice-basis description applies to 
Kagome type non-Bravais lattices as well. In general, 
the 120° AF state is characterized by an ordering plane 
(normal hi) and a planar direction (h,2) in spin space, 
with reference to which spin orientations are given by 



s0a fi2 + sin0 Q (ni x n 2 ) 



(2) 



corresponding to angles 4> a = 0°, 120°, and 240° on the 
three sublattices a = A, B, C. A convenient choice is 
hi — z (spin-ordering in the x — y plane) and fi2 = x, so 
that the local mean field A Q = |j7(*ftr* i ) a in the 120° 
ordered Hartree-Fock state is given by 



Ad (d = a, b, c) 



(3) 



on the three sublattices in terms of the three lattice unit 
vectors 



a = x, 



1, 
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FIG. 1: Brillouin zone (BZ) for the triangular lattice and 
magnetic Brillouin zone (MBZ) for the 120° ordered AF state, 
along with the BZ - MBZ correspondence involving the branch 
index I. 



Fourier transformation within the sublattice basis 
yields 
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where = (a^ a^i 6k| Ckt c ki) m terms of the 
fermion operators &ic,&kjCk defined on the three sub- 
lattices A, B, C. Wavevector k lies within the Mag- 
netic Brillouin Zone (MBZ), corresponding to the three 
inter-penetrating triangular sublattices (lattice parame- 
ter v3a). The NN hopping term 



6 k = -t J2 



e 4k - 5 = - t [e ik *+2e- ik * /2 cos{V3k y /2)}(6) 



mixes AB, BC, and CA sublattices, which are connected 
by the three lattice unit vectors. The lattice hopping 
term 5k yields the triangular-lattice free-fermion energy 



«k = 4 



and transforms as 



5k±Q 



(7) 



(8) 



under momentum translation by the ordering wavevector 
Q = (27r/3,27r/v / 3). 

The [6 x 6] Hamiltonian matrix obeys the cyclic prop- 
erty [Hyw]ab = [Uyw]bc = [Hhf]ca of the 120° ordered 
state, resulting in the following spin-sublattice structure 
of the normalized eigenvectors 
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FIG. 2: (color online) Comparison of density of states cor- 
responding to the full HF dispersion and the direct hopping 
dispersion, showing the band broadening (narrowing) due to 
the virtual hopping dispersion. 



where the planar spin orientations <\> a = 0°, 120°, 240° 
for the three sublattices, and the sublattice phase angle 
Bi = 2irl/3 for the three fermion branches I — 0, ±1. Sub- 
stituting the above eigenvector structure, and contract- 
ing over the sublattice sector a, the eigenvalue equations 
of the [6 x 6] Hamiltonian matrix reduce to three [2 x 2] 
equations in the spin sector corresponding to I = 0, ±1; 
the six eigenvalues and amplitudes (aj ; P^i) ^ or ^ ne 



lower (— ) and upper (+) AF bands are then given by 

(10) 
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where ry k = (ek + £k-q)/2 and £ k = (ek - £k-q)/2, with 
momentum values k, k ± Q corresponding to fermion 
branches £ = 0,±1, respectively. The mean field A 
and magnetization m are related to £/ through the self- 
consistency condition £ 

Typical of the AF state, upper band states yield a 
negative contribution to spin densities due to the neg- 
ative sign of aj. Consequently, when the two bands 
start overlapping on decreasing U , partial occupation of 
the upper band has an amplified effect on reduction of 
magnetic order — reduced sublattice magnetization due 
to band overlap decreases the mean-field A, which fur- 
ther increases the overlap. Typically, the magnetic order 
therefore falls very rapidly after band overlap sets in. 

The density of states (DOS) corresponding to the full 
HF (classical) dispersion in Eq. (10), shown in Fig. 2 for 
both positive and negative t, exhibits a key competition 
between the two dispersion terms associated with direct 
hopping (?7k) of order t and virtual hopping (a/ A 2 + £? — 
A), which is of order J in the strong-coupling limit. For 
positive i, both dispersion terms favour same state at the 



Vk-Q 



(13) 



top of the lower band, while they favour different states 
at the bottom of the upper band, thereby causing band 
narrowing. The competition results in broadening and 
narrowing of the lower and upper bands, depending on 
the sign of t, which has a dramatic effect on the self- 
energy correction due to significantly different density of 
fermion scattering states. There is no such competition 
in the t — J model studies where the virtual hopping 
dispersion term is absent. 
From Eq. (8), it follows that 

«k = -(ek+Q + £k-q) (12) 
so that 

£k 

2 

Therefore, the direct hopping dispersion ?7k-Q, to which 
the HF quasiparticle dispersion (10) reduces in the 
strong-coupling limit, is identical to the classical- level 
dispersion £k/2 taken in earlier t — J model studies, cor- 
responding to effective hopping icos(120°) = — 1/2 asso- 
ciated with the 120° ordering, where momentum transla- 
tion by Q connects the real and slave fermions>i£ Com- 
parison of DOS corresponding to the full dispersion with 
that of yyic is also shown in Fig. 2. 



III. TRANSVERSE SPIN FLUCTUATIONS 

Including both transverse and longitudinal spin 
fluctuations, the full spin-fluctuation propagator 
(V G \T[S?(t)S% (?)}•>& G ) in the 120° ordered AF state 
has been studied recently in the random phase ap- 
proximation (RPA) in the full U range. 9 Even in the 
intermediate-coupling regime, the magnitude of longitu- 
dinal fluctuation (S 2 ,) along the local ordering directions 
a was found to be quite negligible ((S^) ~ 10~ 4 
at U « 7). We therefore focus on transverse spin 
fluctuations along the two locally normal spin directions. 

Spin rotation about the z direction 
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by angles <j) a = 0°, 120°, -120° for a = A,B,C renders 
x' as the spin ordering direction for all three sublattices. 
In the 2 <$ 3 spin-sublattice basis of the two transverse 
spin directions fi, v — y', z' and the three sublattices 
a,(3 — A,B,C, the RPA-level spin-fluctuation propaga- 
tor is then given by 

' 1- J7[x°(q,w)] 
where the bare particle-hole propagator 
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involves integrating out the fermions in the broken- 
symmetry state. In the particle-hole matrix elements of 
the rotated spins 



(17) 



the spin orientation angles <j) a in the fermion states (Eq. 
9) are transformed out. 

We now discuss the spin-sublattice structure of the 
[x°(q, uj)\ matrix and its eigenvectors. While the spin- 
diagonal blocks [x°(q, u>)]w are Hcrmitian and the off- 



diagonal blocks [x (q, £ 



are anti-Hcrmitian, they 



obey the cyclic symmetry [x°(<1,u)]ab = [x°(q,^)]sc = 
[x°(q, cj)]ca of the 120° ordered phase. Also, the sub- 
lattice diagonal elements [x°(q> w)]aa are all identical 
due to sublattice symmetry. Consequently, the normal- 
ized eigenvectors of [x°(q, w)] have the following spin- 
sublattice structure 
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where A = 0, ±1 for the three magnon branches, and 
the real and normalized amplitudes u and v represent 
the fluctuation amplitudes in the y' and z' directions, 
respectively. Contracting over the sublattice index, the 
eigenvalue equation for \4>) therefore reduces to 



[ X (q, W )M A = [x° A (q, W )] 



= A r 



(19) 



where [x°(q, to)] is a [2 x 2] real-symmetric matrix. 

Solving the pole equation 1 — UXq(oj) = for the 
magnon energy w q , and expanding A q (w) around the 
poles to obtain the magnon amplitudes, yields the 
magnon propagator 
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where the magnon eigenvectors for the advanced (A) and 
retarded (R) modes are given by 
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in terms of the magnon amplitudes 

^q,A = u^\/^j2U 2 \d\^\/<ko\ and 
= Vq.x/ ^2U 2 \dX^x/du!\ 



(22) 



in the y' and z' directions. Expressions for magnon 
energy and amplitudes in the strong-coupling limit are 
given in the Appendix. 
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FIG. 3: Self energy in the noncrossing approximation. Wavy 
lines represent the magnon propagator. The bare fermion- 
magnon interaction vertex is U. 



IV. SELF-ENERGY CORRECTION 

Due to multiple magnon emission and absorption pro- 
cesses associated with fermion motion in the AF state, 
the fermion self energy matrix in the spin-sublattice basis 
is obtained as 

[£(M,a;)U 

= U2 J |^EM[G«Mk-q,m, W -Q)Mx(q,tt)]: 



(22 



in the non-crossing (rainbow) approximation. 

We consider the intraband contribution involving hole 
(electron) scattering states in the lower (upper) band, 
which redistributes the spectral function, leaving the in- 
tegrated spectral weight and the sublattice magnetiza- 
tion unchanged. We obtain for the hole self energy 



£ k ,,(w) = (k,i|[S(k, 
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(24) 

where the fermion-magnon scattering matrix element 

m = <y E< k > 'Kl k - q> m )« ® |q. A >^ ( 25 ) 



involves the advanced magnon mode. Substituting the 
sublattice structure of the fermion and magnon ampli- 
tudes, the sum over sublattice index a yields 

l (26) 



1 + e «( m + A -')27r/3 _|_ e -»(m+A-J)27r/3 _ 3^ 



effectively amounting to a conservation of sublattice 
polarization at the fermion-magnon interaction vertex. 
Therefore, the fermion-magnon scattering matrix ele- 
ment reduces to a sum of the y' and z' fluctuation terms 

M= ~f[ - ( a k,j/ 3 k-q,m -^k,J a k-q,m)yq.A 
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FIG. 4: Variation of hole self energy with iterations, effec- 
tively illustrating the role of successively higher-order magnon 
processes. Here A = 4 (U — 8.8) and k = 0, I = — 1. 



For fermion and magnon states in the matrix element 
M, the sublattice-basis MBZ description of momentum 
translates to a BZ description according to the corre- 
spondence k, I — > k + and q, A — » q + AQ. With this 
equivalence, Eq. (26) simply corresponds to momentum 
conservation in the BZ. Analysis of the fermion-magnon 
matrix element M in the strong-coupling limit and com- 
parison with earlier results for the t—J model is discussed 
in the Appendix. 

It is important to note here that long-wavelength 
magnon modes yield finite contribution to the fermion- 
magnon scattering process in the triangular-lattice AF, 
unlike the square-lattice case where this contribution is 
negligible^ For the square-lattice AF, the small-g con- 
tribution was suppressed because the fermion-magnon 
matrix element M 2 ~ q due to destructive interference 
within sublattice summation. For the triangular lat- 
tice also, for q = and A = (in-plane mode), the 
fermion matrix element in Eq. (25) reduces to an ex- 
pectation value which identically vanishes for fj, = y' as 
the spins are oriented in the x' direction, yielding sim- 
ilar M 2 ~ q behaviour for small q. However, for the 
out-of-plane z' fluctuation modes (A = ±1), the fermion 
matrix element is finite, resulting in M 2 ~ 1/q and a 
finite contribution of long-wavelength modes within the 
two-dimensional (J qdq) momentum summation. 
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FIG. 5: (color online) Quasiparticle dispersion along dif- 
ferent symmetry directions in the MBZ (lower panel), along 
with the HF dispersion for comparison (upper panel). 



lower band (Fig. 4) and fifteen iterations for the upper 
band (Fig. 8). 

Self-energy corrections for an added hole in the broad, 
flat band near the top of the lower band are qualitatively 
different from that of an added electron in the narrow, 
sharp peak near the bottom of the upper band. The low 
density of hole scattering states and the dominant band- 
energy denominator suppresses the hole self energy. How- 
ever, the electron self energy is significantly enhanced due 
to the sharp peak and the small band energy compared 
with magnon energy, resulting in the characteristic signa- 
ture of string states associated with multi-magnon pro- 
cesses. It is convenient to visualize these qualitatively dif- 
ferent self-energy corrections in terms of hole (electron) 
motion in an effective ferromagnetic (antiferromagnetic) 
spin background projected out of the 120° spin ordering. 



V. RESULTS AND DISCUSSION 

The self-consistent numerical evaluation of the self en- 
ergy (24) was carried out on a 30 x 40 grid in the MBZ 
k space and a frequency interval Alu = 0.025 for to in 
the range —15 < to < 10. The self energy was itera- 
tively evaluated, starting with Sk(w) = 0. Typically, 
self-consistency was achieved within ten iterations for the 



A. Hole Dynamics (Lower Band) 

Figure 5 shows renormalized quasiparticle dispersion 
along different symmetry directions in the MBZ. Com- 
parison with the HF result shows nearly momentum in- 
dependent shift in the quasiparticle energies, leaving the 
effective hole mass essentially unchanged. States in up- 
per portion of the band are pushed up, while those in the 
lower portion are pulled down, in accordance with the 
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FIG. 6: (color online) Hole spectral function Au.(uj) 
special MBZ points V, K, and M. Here A = 4. 



formally second-order structure of the self-energy correc- 
tion. The quasiparticle (hole) energy is maximum (mini- 
mum) at the r point k = (0, 0) for / = — 1, corresponding 
to BZ momentum — Q. 

Figure 6 shows the spectral function Ak(ui) for the spe- 
cial points r, K, and M. The weak self-energy correction 
in the broadened lower band of the frustrated 120° or- 
dered AF state results in no visible oscillatory structure, 
typically associated with the string of broken bonds as 
the hole moves in the AF background. As expected, the 
spectral function at the T point k = (0,0) and I = —1 
shows a coherent quasiparticle peak, as this state lies at 
the top of the lower band. All three branches are degener- 
ate at point K, as at the HF level. The spectral functions 
at K and M points show well-defined peak structures, 
with finite quasiparticle damping as these states lie well 
within the band. The quasiparticle peak broadens and 




FIG. 7: 

r - K 

I = -1. 



(color online) Hole spectral function Ak(w) along the 
and F — M directions for the lowest-energy branch 



loses intensity along both the Y—K and T—M directions, 
as seen in Fig. 7. 



B. Electron Dynamics (Upper Band) 

Self-energy correction in the narrow upper band is rela- 
tively large and self consistency is noticeably slower (Fig. 
8) , illustrating the importance of multi-magnon processes 
and resulting in the characteristic oscillatory structure in 
the spectral functions (Fig. 10) associated with string 
states. 

Figure 9 shows the quasiparticle dispersion along the 
r — K — M — r directions for the lowest-energy branch 
1 = 0. The lowest-energy state at point M shows a well- 
defined quasiparticle peak at ui = 1.3, along with a long 
incoherent tail, as seen in Fig. 10. All branches are 
degenerate at K, second and third branches at M, first 
and second branches at T, while at N = (2tt/5,0) all 
branches are non-degenerate, exactly as at the HF level. 

The states K, T, and N, which are well inside the 
band, are strongly damped and yield dominantly inco- 
herent spectral functions, along with small quasiparticle 
peaks at nearly same frequency u> ~ 2 (Fig. 10). It is 
the strong negative peak in the self energy at ui 2 (Fig. 
8) which leads to nearly same quasiparticle energy for all 
k points, resulting in drastically reduced quasiparticle 
bandwidth (Fig. 9) and enhanced effective mass. Fig- 
ure 11 shows that the well-defined quasiparticle peak at 
M rapidly diminishes in intensity and disappears as one 
moves along the M — T direction. However, in the M — K 
direction, the quasiparticle peak is discernible in the full 
k range. In the K — V direction, the quasiparticle peak 
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FIG. 8: Variation of electron self energy with iterations for 
the upper band, effectively illustrating the role of succes- 
sively higher-order magnon processes. Here A = 4 and 
k = (-2f,0),J = 0. 
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FIG. 9: (color online) Quasiparticle dispersion E^. for the 
lowest-energy branch I = along different symmetry direc- 
tions in the MBZ (lower panel), along with the HF dispersion 
for comparison (upper panel). 
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FIG. 10: (color online) Electron spectral function Ak(w) at 
the special MBZ points for A = 4 (U ~ 8.8). 



marginally increases and then rapidly disappears as one 
moves towards T. 

The lowest-energy hole and electron pockets are shown 
in Fig. 12. The one-particle density of states is shown 
in Fig. 13 for A = 4 and 3. The classical-level asym- 
metry strongly influences the quantum corrections and 
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FIG. 11: (color online) Evolution of the electron spectral func- 
tion for the lowest-energy branch I = along different sym- 
metry directions. 




FIG. 12: (color online) Hole (blue) and electron pockets (red) 
corresponding to lowest-energy states in the Brillouin zone. 
Qi, Q2, Q3 are equivalent AF ordering wave vectors. 



the characteristic signature of multi-magnon processes is 
dominant for the upper band corresponding to the nar- 
row sharp peak at the classical level. As expected within 
the many-body expansion, the string-state signature of 
multi-magnon processes is more prominent for higher U. 
The renormalized gap vanishes for U « 7, whereas the 
HF band gap 2 (A— 2) vanishes at U ~ 5. The band gap is 
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FIG. 13: (color online) Renormalised density of state for one 
added hole and electron for (a) A = 4 and (b) A = 3, showing 
the vanishing of energy gap. 



indirect, with the lowest-energy hole and electron states 
corresponding to momenta (47r/3,0) and (— 27r/3, 0), re- 
spectively. 

It is interesting to note that in the intermediate- 
coupling regime of interest, the interaction strength ef- 
fectively controls the fermion-magnon scattering through 
three different aspects. With decreasing interaction 
strength, besides the explicit reduction in the fermion- 
magnon scattering matrix element due to U in Eq. (25), 
the classical-level fermion and magnon dispersions are 
also significantly modified. Due to enhanced virtual hop- 
ping and competition with the direct hopping term the 
sharp peak in the upper band DOS (Fig. 2) shifts towards 
higher energy. Also, the magnon energy com at momen- 
tum Q/2 decreases rapidly and vanishes at A = 2. 9. 9 
Fig. 13 shows the modifications in the upper band DOS 
due to these intermediate-t/ effects. 



VI. CONCLUSIONS 

In conclusion, we have studied the hole and electron 
dynamics in the 120° ordered AF state of the Hubbard 
model on a triangular lattice using a physically transpar- 
ent fluctuation approach involving the dynamical spin 
fluctuations, which interpolates between the weak and 
strong coupling limits. Finite-?/, double occupancy ef- 



9 



fects, neglected in earlier t — J model studies, have been 
incorporated naturally in terms of classical level fermion 
and spin dynamics. Intrinsic features of the frustration- 
induced direct hopping dispersion associated with the 
120° ordering - the broad flat band and the narrow sharp 
peak in the fermion DOS corresponding to lowest-energy 
hole and electron states - are characteristics of ferromag- 
netic and antiferromagnetic ordering, respectively. The 
qualitatively different self-energy corrections for hole and 
electron can therefore be conveniently visualized in terms 
of hole (electron) motion in an effective ferromagnetic 
(antiferromagnetic) spin background projected out of the 
120° spin ordering. 

For an added hole in the broad lower band, the re- 
duced density of scattering states suppresses the fermion- 
magnon interaction resulting in nearly coherent quasipar- 
ticle peak for all k states. No signature of string states in 
the spectral function reflects an effective F background 
seen by holes near the top of the lower band. Quasipar- 
ticle dispersion shows a nearly momentum-independent 
shift of hole energies, implying no mass renormalization. 

For an electron in the lowest-energy branch I = of 
the upper band, quasiparticle peak is observed only near 
the MBZ boundary (M — K) , and rapidly vanishes away 
from it. Strong incoherent behaviour and clear signa- 
ture of string states in the spectral function is a conse- 
quence of an effective AF background seen by electrons 
near the bottom of the upper band. The strong and 
nearly momentum-independent peak in the self energy 
leads to nearly same quasiparticle energy for all k points, 
resulting in drastically reduced bandwidth and enhanced 
effective mass. 

The renormalized band gap was found to vanish for 
U rs 7, yielding a first-order M-I transition, as also ob- 
tained earlier for the frustrated square- lattice AFil On 
the other hand, for the unfrustrated AF the band gap 
never vanishes for any finite U. The vanishing of band 
gap at moderate U for both frustrated antiferromagnets 
due to frustration-induced band broadening thus high- 
lights the role of frustration in M-I transition. 

Finally, frustration and spin fluctuations are involved 
in an interesting interplay with respect to stability of 
the insulating state. Frustration generally enhances spin 
fluctuations and magnetic disordering. It will there- 
fore be interesting to also examine the interband self- 
energy contribution which reduces magnetic order due 
to interband spectral weight transfer, and also widens 
the band gap and thereby stabilizes the insulating state. 
The first-order interband contribution exactly cancels for 
negligible AF bandwidth, as for the unfrustrated AF in 
the strong-coupling limit ^ but will survive for finite 
frustration-induced bandwidth, thus highlighting the in- 
terplay between frustration and spin fluctuations, and 
providing deeper insight into the nature of the Mott in- 
sulator. 



APPENDIX: FERMION-MAGNON MATRIX 
ELEMENT IN THE STRONG-COUPLING LIMIT 

We show that to leading order in the U/t — > oo limit, 
the intraband fermion matrix elements reduce to order 
t/ A, and the fermion-magnon matrix element explicitly 
reduces to order i, as within the t — J model. 

For the magnon energy and magnon amplitudes in the 
y' and z' directions we have 



yl = 



zi = 



3JS[(l- 7q )(l + 2 7q )] 1/2 
MS,-, ~ \ 

■(l+2 7 q) 



where 



7q 



4w q 
3JS 



4w n 

1 

3 



(l"7q) 



(A.l) 



1x a/3 
cos q x + 2 cos — cos ~^~1y 



For the lower (— ) and upper (+) bands, the fermion 
amplitudes given in Eq. (11) reduce to 



±4= (i±lr 

V2 V 2A 



PI 



"71 : &) — 



Hence in terms of the spin raising and lowering operators 
defined below, the intraband fermion matrix elements 



= (k, l\a z i +ia y '\k- q, m) 



ia y '\k-q, m) 



6c,i 

A 

£k-q,'. 



for lower band states are explicitly of order t/U. 
In terms of corresponding magnon amplitudes 

$ q,A = V^[|q, ± i|q, \) y >] = Z q , A ± ^ q , A 



(A.3) 



(A.4) 



for the advanced mode, the fermion-magnon matrix ele- 
ment 

V3M = U[-i{<ry)y+ (<J Z ,)Z] 



U[(a+)*- + <0« + ]/2 



-(£k,Z _ Ck-q.m^q.A + (£k,Z + £k-q,m)-Zq,A 



6c,Z$, 



q,A 



6c- 



q,™^q : A 



(where m — I — A) 
(A.5) 



explicitly reduces to order t. 

The above expression for the fermion-magnon ma- 
trix element has exactly same structure as the result 
y3tM (k, q) — VStlvqhk — it q /ik+ q ] obtained within the 
t — J model. 17 Indeed, magnon amplitudes it q and w q in 
Ref. [17] exactly correspond to i? q ± 3^q, and (V3t)h^ = 
V3tJ2$ s i n k-5 = — Ck-Q- However, the Goldstone- 
mode contribution appears to be different. While our 
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fermion-magnon matrix element vanishes for the Gold- 
stone mode q — > 0, A = 0, for which 7 q — * 1 and the 
magnon amplitude Z — > representing rigid spin rota- 
tion about z axis, the matrix element M(k, q) in Ref. 
[17] does not vanish. The other two Goldstone modes 
q -> 0, A = ±1, for which y q -> -1/2 and y -> 0, 



do yield non- vanishing matrix elements with A/ 2 ~ l/q, 
although resulting in a finite contribution J qdq/q from 
long-wavelength modes. Thus, for the triangular-lattice 
AF, long-wavelength modes do contribute to the fermion- 
magnon scattering. 
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